Memory retention system

ABSTRACT

The present disclosure generally relates to a computer-implemented system for intelligently retaining and recalling memory data. An exemplary method comprises obtaining a speech input of a user; obtaining a text input of the user; constructing a first instance of a memory data structure based on the speech input; constructing a second instance of the memory data structure based on the text input; adding the first instance and the second instance of the memory data structure to a memory stack of the user; obtaining a message from a sender to the user; retrieving a particular instance of the memory data structure from the memory stack based on the message; generating a response to the message based on the retrieved particular instance of the memory data structure; and causing display of the response to the message in a messaging user interface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/185,572, filed on Feb. 25, 2021, which claims the benefit of U.S.Provisional Application 62/983,464 filed on Feb. 28, 2020, the entirecontents of which are incorporated herein by reference in theirentirety.

FIELD OF THE INVENTION

The present disclosure relates generally to a memory retention system,and more specifically to techniques for automatically constructing asearchable electronic version of human memory and for providing anatural retrieval user interface (i.e., a recall user interface).

BACKGROUND

Human memory is the process in which information is encoded, stored, andretrieved in the brain. Human memory, however, can be fickle. A majorityof the population has concerns about retention of short-term and/orlong-term memory. Thus, there is a need for a memory retention systemaugmenting human's biological memory and recall functions.

BRIEF SUMMARY

An exemplary computer-implemented method comprises: receiving, via amicrophone of an electronic device, a speech input of the user;receiving, via the electronic device, a text input of the user;constructing a first instance of a memory data structure based on thespeech input, wherein the first instance comprises a transcript of thespeech input and is associated with a first set of context parameters;constructing a second instance of the memory data structure based on thetext input, wherein the second instance comprises at least a portion ofthe text input and is associated with a second set of contextparameters; adding the first instance and the second instance of thememory data structure to a memory stack of the user; displaying a userinterface for retrieving memory data of the user; receiving, via theuser interface, a beginning of a statement from the user; retrieving aparticular instance of the memory data structure from the memory stackbased on the beginning of the statement; and automatically displaying acompletion of the statement based on the retrieved particular instanceof the memory data structure.

In some embodiments, the user interface is invoked by a predefinedcombination of simultaneous key presses.

In some embodiments, the user interface includes a user interfacecontrol for adding another user's memory stack.

In some embodiments, the method further comprises: responsive to a userinput, displaying the particular instance of the memory data structurein a timeline interface.

In some embodiments, the method further comprises receiving a user inputto modify the particular instance of the memory data structure.

In some embodiments, the timeline interface identifies one or moreinstances of memory data structure that are related to the particularinstance.

In some embodiments, the text input comprises an input in a messagingapplication, an email application, or a collaboration application.

In some embodiments, the method further comprises obtaining the firstset of context parameters associated with the speech input.

In some embodiments, the first set of context parameters comprises oneor more entities extracted from the speech input.

In some embodiments, the first set of context parameters comprises oneor more emotions extracted from the speech input.

In some embodiments, the first set of context parameters comprises asummary of the speech input.

In some embodiments, the first set of context parameters comprises aconcept extracted from the speech input.

In some embodiments, the first set of context parameters is obtainedfrom a third-party application.

In some embodiments, the third-party application is a calendarapplication.

In some embodiments, the memory stack is stored in a blockchain-baseddatabase.

In some embodiments, the completion of the statement comprises a time, aperson, a location, a concept, or any combination thereof.

The method of claim 1, further comprising: receiving, via the electronicdevice, an image; constructing a third instance of the memory datastructure based on the image; and adding the third instance of thememory data structure to the memory stack.

The method of claim 1, further comprising: receiving data from one ormore sensors of the electronic device; constructing a fourth instance ofthe memory data structure based on the received data; and adding thefourth instance of the memory data structure to the memory stack.

An electronic device comprises: one or more processors; a memory; andone or more programs, wherein the one or more programs are stored in thememory and configured to be executed by the one or more processors, theone or more programs including instructions for: receiving, via amicrophone of an electronic device, a speech input of the user;receiving, via the electronic device, a text input of the user;constructing a first instance of a memory data structure based on thespeech input, wherein the first instance comprises a transcript of thespeech input and is associated with a first set of context parameters;constructing a second instance of the memory data structure based on thetext input, wherein the second instance comprises at least a portion ofthe text input and is associated with a second set of contextparameters; adding the first instance and the second instance of thememory data structure to a memory stack of the user; displaying a userinterface for retrieving memory data of the user; receiving, via theuser interface, a beginning of a statement from the user; retrieving aparticular instance of the memory data structure from the memory stackbased on the beginning of the statement; and automatically displaying acompletion of the statement based on the retrieved particular instanceof the memory data structure.

An exemplary non-transitory computer-readable storage medium stores oneor more programs, the one or more programs comprising instructions,which when executed by one or more processors of an electronic devicehaving a display, cause the electronic device to: receive, via amicrophone of an electronic device, a speech input of the user; receive,via the electronic device, a text input of the user; construct a firstinstance of a memory data structure based on the speech input, whereinthe first instance comprises a transcript of the speech input and isassociated with a first set of context parameters; construct a secondinstance of the memory data structure based on the text input, whereinthe second instance comprises at least a portion of the text input andis associated with a second set of context parameters; add the firstinstance and the second instance of the memory data structure to amemory stack of the user; display a user interface for retrieving memorydata of the user; receive, via the user interface, a beginning of astatement from the user; retrieve a particular instance of the memorydata structure from the memory stack based on the beginning of thestatement; and automatically display a completion of the statement basedon the retrieved particular instance of the memory data structure.

An exemplary computer-implemented method for retaining and retrieving amemory of a user comprises: receiving an audio input comprising speechdata; deriving, based on the audio input, text, visuals, and/or locationdata, a plurality of parameters; generating a natural-language textstring based on the plurality of parameters; associating thenatural-language text string with the plurality of parameters; storingthe natural-language text string in a time-series database; receiving aquery of the user; retrieving, based on the query, the natural-languagetext string; and outputting the retrieved natural-language text string.

In some embodiments, a number of words in the natural-language textstring is lower than a predefined maximum value.

In some embodiments, the audio input is received via a microphone of awearable device.

In some embodiments, the query is received via the microphone of thewearable device.

In some embodiments, the plurality of parameters comprise: a timeparameter, a location parameter, an entity parameter, a peopleparameter, a context parameter, an emotion parameter, a visual memoryparameter, or any combination thereof.

In some embodiments, the method further comprises receiving a set of GPScoordinates; obtaining a location parameter from the set of GPScoordinates; and associating the natural-language text string with thelocation parameter.

In some embodiments, the method further comprises receiving an image;obtaining one or more parameters from metadata of the image; andassociating the one or more parameter with the natural-language textstring.

In some embodiments, the method further comprises receiving social mediadata of the user; obtaining one or more parameters from the social mediadata; and associating the one or more parameter with thenatural-language text string.

In some embodiments, the method further comprises storing thenatural-language text string in a graph-based database.

In some embodiments, the query of the user is a text string.

In some embodiments, the query of the user comprises an audio input andan eye blink from the user.

An exemplary method for retaining and retrieving a memory of a usercomprises: at an electronic device having a microphone and an eyemovement detection mechanism: detecting, via the eye movement detectionmechanism, an eye movement; determining whether the eye movement meets apredefined requirement; if the eye movement meets the predefinedrequirement, formulating one or more parameters of a query based on anaudio input received by the microphone in a predefined time periodpreceding the detected eye movement.

In some embodiments, the electronic device comprises a pair of glasses.

In some embodiments, the eye movement detection mechanism comprises aplurality of electrodes affixed to the electronic device.

In some embodiments, the eye movement comprises an eye blink.

In some embodiments, the predefined requirement comprises a minimumduration of the eye movement.

An exemplary system for retaining and retrieving a memory of a user,comprises: one or more processors; a memory; and one or more programs,wherein the one or more programs are stored in the memory and configuredto be executed by the one or more processors, the one or more programsincluding instructions for: receiving an audio input comprising speechdata; deriving, based on the audio input, a plurality of parameters;generating a natural-language text string based on the plurality ofparameters; associating the natural-language text string with theplurality of parameters; storing the natural-language text string in atime-series database; receiving a query of the user; retrieving, basedon the query, the natural-language text string; and outputting theretrieved natural-language text string.

An exemplary non-transitory computer-readable storage medium stores oneor more programs, the one or more programs comprising instructions,which when executed by one or more processors of a system, cause thesystem to: receive an audio input comprising speech data; derive, basedon the audio input, a plurality of parameters; generate anatural-language text string based on the plurality of parameters;associate the natural-language text string with the plurality ofparameters; store the natural-language text string in a time-seriesdatabase; receive a query of a user; retrieve, based on the query, thenatural-language text string; and output the retrieved natural-languagetext string.

An exemplary electronic device for retaining and retrieving a memory ofa user, comprises: a microphone; an eye movement mechanism; one or moreprocessors; a memory; and one or more programs, wherein the one or moreprograms are stored in the memory and configured to be executed by theone or more processors, the one or more programs including instructionsfor: detecting, via the eye movement detection mechanism, an eyemovement; determining whether the eye movement meets a predefinedrequirement; if the eye movement meets the predefined requirement,formulating one or more parameters of a query based on an audio inputreceived by the microphone in a predefined time period preceding thedetected eye movement.

An exemplary non-transitory computer-readable storage medium stores oneor more programs, the one or more programs comprising instructions,which when executed by one or more processors of an electronic devicehaving a microphone and an eye movement mechanism, cause the system to:detect, via the eye movement detection mechanism, an eye movement;determine whether the eye movement meets a predefined requirement; ifthe eye movement meets the predefined requirement, formulate one or moreparameters of a query based on an audio input received by the microphonein a predefined time period preceding the detected eye movement.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates exemplary user interfaces of a memory retentionsystem providing memory retention and retrieval, in accordance with someembodiments.

FIG. 1B illustrates an exemplary user interface of a memory retentionsystem providing memory retention and retrieval, in accordance with someembodiments.

FIG. 1C illustrates an exemplary user interface of a memory retentionsystem providing memory retention and retrieval, in accordance with someembodiments.

FIG. 1D illustrates an exemplary user interface of a memory retentionsystem providing memory retention and retrieval, in accordance with someembodiments.

FIG. 2A illustrates an exemplary process for generating a memorysegment, in accordance with some embodiments

FIG. 2B illustrates parameters in an exemplary memory segment and thesources from which each parameter can be generated, in accordance withsome embodiments.

FIG. 2C illustrates an exemplary graph-based database, in accordancewith some embodiments.

FIG. 3A illustrates an exemplary process 300 for retrieving a memorysegment or a memory segment portion, in accordance with someembodiments.

FIG. 3B illustrates exemplary user interfaces of a memory retentionsystem providing memory retention and retrieval, in accordance with someembodiments.

FIG. 3C illustrates exemplary user interfaces of a memory retentionsystem providing memory retention and retrieval, in accordance with someembodiments.

FIG. 3D illustrates exemplary user interfaces of a memory retentionsystem providing memory retention and retrieval, in accordance with someembodiments.

FIG. 4A illustrates exemplary electrooculography techniques of a memoryretention system, in accordance with some embodiments.

FIG. 4B illustrates exemplary electrooculography techniques of a memoryretention system, in accordance with some embodiments.

FIG. 4C illustrates exemplary electrooculography techniques of a memoryretention system, in accordance with some embodiments.

FIG. 5 depicts an exemplary electronic device in accordance with someembodiments.

FIG. 6A illustrates exemplary data structures for storing memory data,in accordance with some embodiments.

FIG. 6B illustrates an exemplary memory retention system for generatingmemory data structures and retrieving memory data structures, inaccordance with some embodiments.

FIG. 7A illustrates an exemplary recall user interface, in accordancewith some embodiments.

FIG. 7B illustrates an exemplary recall user interface, in accordancewith some embodiments.

FIG. 7C illustrates an exemplary recall user interface, in accordancewith some embodiments.

FIG. 7D illustrates an exemplary recall user interface, in accordancewith some embodiments.

FIG. 7E illustrates an exemplary recall user interface, in accordancewith some embodiments.

FIG. 8A illustrates an exemplary user interface of the memory retentionsystem, in accordance with some embodiments.

FIG. 8B illustrates an exemplary user interface of the memory retentionsystem, in accordance with some embodiments.

FIG. 8C illustrates an exemplary user interface of the memory retentionsystem, in accordance with some embodiments.

FIG. 9 illustrates an exemplary recall user interface within anintegrated third-party application, in accordance with some embodiments.

FIG. 10 illustrates an exemplary recall user interface within anintegrated third-party application, in accordance with some embodiments.

DETAILED DESCRIPTION

The present disclosure is directed to methods, systems, devices,apparatus, and non-transitory storage media for constructing anelectronic version of human memory and providing easy retrieval of thememory. Embodiments of the present disclosure can receive memory data(e.g., a continuous stream of audio data) of a user, automaticallyconstruct memory segments or data structures that capture rich contentand context of the user's life experiences, store the memory segments,and provide user interfaces for easy and natural retrieval of the memorysegments. Embodiments of the present disclosure can be deployed viawearable devices, mobile devices, digital assistants, computers, or anycombination thereof.

Embodiments of the present disclosure can augment human's biologicalmemory and recall functions. Embodiments of the present disclosure canadvantageously facilitate recall of both short-term and long-termmemories for a user. Embodiments of the present disclosure cansupplement or provide efficient functioning memory for people with highknowledge burden, daily functioning memory for people with performanceissues (such as memory loss), and enhanced functioning memory for ahuman being with average retention.

The following description sets forth exemplary methods, parameters, andthe like. It should be recognized, however, that such description is notintended as a limitation on the scope of the present disclosure but isinstead provided as a description of exemplary embodiments.

Although the following description uses terms “first,” “second,” etc. todescribe various elements, these elements should not be limited by theterms. These terms are only used to distinguish one element fromanother. For example, a first graphical representation could be termed asecond graphical representation, and, similarly, a second graphicalrepresentation could be termed a first graphical representation, withoutdeparting from the scope of the various described embodiments. The firstgraphical representation and the second graphical representation areboth graphical representations, but they are not the same graphicalrepresentation.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

The term “if” is, optionally, construed to mean “when” or “upon” or “inresponse to determining” or “in response to detecting,” depending on thecontext. Similarly, the phrase “if it is determined” or “if [a statedcondition or event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event],” depending on the context.

FIGS. 1A-1D illustrates exemplary user interfaces of a memory retentionsystem providing memory retention and retrieval, in accordance with someembodiments. In the example depicted in FIG. 1A, an exemplary user Erinand her friend Rachel met for dinner at Las Havas restaurant at t₁(i.e., 8 pm on Feb. 18, 2020). Aspects of Erin's conversation withRachel are captured and retained by the system. At t₂ (i.e., the nextday), Erin has a conversation with Kristie about her dinner with Rachel,and is able to recall aspects of the conversation with the help of thememory retention system via a natural recall user experience. The userinterfaces are described below with reference to process 200 (FIG. 2A)and process 300 (FIG. 3A).

FIG. 2A illustrates an exemplary process 200 for generating a memorysegment or data structure, in accordance with some embodiments. Process200 is performed, for example, using one or more electronic devicesimplementing a software platform. In some examples, process 200 isperformed using a client-server system, and the blocks of process 200are divided up in any manner between the server and a client device. Inother examples, the blocks of process 200 are divided up between theserver and multiple client devices. Thus, while portions of process 200are described herein as being performed by particular devices of aclient-server system, it will be appreciated that process 200 is not solimited. In other examples, process 200 is performed using only a clientdevice (e.g., wearable device 102) or only multiple client devices. Inprocess 200, some blocks are, optionally, combined, the order of someblocks is, optionally, changed, and some blocks are, optionally,omitted. In some examples, additional steps may be performed incombination with the process 200. Accordingly, the operations asillustrated (and described in greater detail below) are exemplary bynature and, as such, should not be viewed as limiting.

With reference to FIG. 2A, an exemplary system (e.g., one or moreelectronic devices) receives memory data 202. Memory data 202 includesdata that captures life experiences of a user. The memory data can beassociated with time information (e.g., time stamps) indicating when thememory data is captured.

In some embodiments, the memory data comprises audio data. The audiodata comprises speech audio from the user, speech audio from others,ambient audio, or a combination thereof.

The audio data can be captured by an input device of the systemconfigured to continuously capture audio data. In some embodiments, theinput device comprises one or more microphone and can be integrated witha wearable device (e.g., glasses 102).

The one or more microphones can include a bone conduction microphone, aMEMS microphone, or any combination thereof. In some embodiments, theinput device comprises multiple microphones for receiving simultaneouslymultiple audio streams (e.g., corresponding to speech and ambient audiorespectively) and allowing for ambient noise cancellation.

In the depicted example in FIG. 1A, at t₁, the feed device of the systemcaptures the conversation between Rachel and Erin at t₁. The audio datacomprises Rachel's speech (“We always come here for date night. Iactually know the chef, Alan, through fencing club—really nice guy. Lasttime we came, he somehow found out I love enchiladas del mar and sent itout to me before I event ordered!”), Erin's speech (“Great!”), andambient audio. The audio data is associated with one or more time stampsindicating when the audio data is captured.

In some embodiments, the memory data comprises location data (e.g., GPScoordinates) of the user. The location data can be captured by awearable device (e.g., the user's glasses 102), a mobile device (e.g.,the user's mobile phone), or a combination thereof. The location data isassociated with one or more time stamps indicating when the locationdata is captured.

In some embodiments, the memory data comprises image or media data suchas photos and videos. The image data can be captured by a wearabledevice (e.g., the user's glasses 102), a mobile device (e.g., the user'smobile phone), or a combination thereof. The image data can be furtherobtained from software applications, software services, and/or databasesassociated with the user. For example, the system can access the user'sphotos from Google Photos. The image data include metadata such aslocation and time stamps indicating where and when the image data iscaptured.

In some embodiments, the memory data comprises social media data. Thesocial media data includes photos, posts, shared content, and contactsof the user. The system can access social media data of the user uponreceiving permission and login credentials from the user. The socialmedia data includes metadata such as location and time stamps.

It should be appreciated that the above-described examples of memorydata are merely exemplary. Memory data 202 can include any data thatcaptures life experiences of the user. For example, any user-specificinformation (e.g., the user's calendar, emails, devices, applications,subscribed services) can be part of the memory data 102, as described indetail herein. Further, information derived from the user-specificinformation (e.g., information of a concert that is in the user'scalendar) can be part of the memory data 202.

At block 204, the system constructs a memory segment 212 based on thememory data 202. In some embodiments, the memory segment comprises aplurality of associated, predefined parameters, including a text string,time, people, entity, context, geolocation, emotion, and visual memory.Each of the predefined parameters can be extracted from the memory data202.

The memory retention system can include an audio processing system 206for processing the audio data in the memory data 202, as describedbelow.

In some embodiments, the audio processing system 206 comprises one ormore speech-to-text algorithms to process audio speech data to obtain atranscription of the speech. In the depicted example in FIG. 1A, theaudio processing system can obtain a transcription of Rachel's speechand Erin's speech. The audio processing system can further associateportions of the transcription (e.g., every syllable, every word, everyphrase, every sentence) with a time stamp. The speech-to-text algorithmscan include open-source algorithms, proprietary algorithms, or anycombination thereof.

In some embodiments, the audio processing system 206 comprises one ormore speaker diarization algorithms to identify speaker identitiesassociated with the transcription. In the depicted example in FIG. 1A,the audio processing system can identify Rachel's speech as beingassociated with a first user identity and Erin's speech as beingassociated with a second identity (e.g., based on the audiocharacteristics of the speech). In some embodiments, the audioprocessing system can determine that the second identity is the user ofthe memory retention system (e.g., based on the audio characteristics ofthe speech). The speaker diarization algorithms can include open-sourcealgorithms, proprietary algorithms, or any combination thereof.

In some embodiments, the audio processing system 206 comprisesnatural-language processing techniques for extracting entities. In thedepicted example, the audio processing system can extract entities suchas “Alan” and “enchiladas del mar” from Rachel's speech. The audioprocessing system can further classify the entities (e.g., classifying“Alan” as a person, classifying “enchiladas del mar” as food). Exemplaryalgorithms can include open-source algorithms (e.g., Rasa Open Source,SpaCy), proprietary algorithms, or any combination thereof.

In some embodiments, the audio processing system 206 comprises entitylinking techniques for establishing associations among entities andbuild a database of the memory retention system. The database caninclude a plurality of entities (e.g., with unique identifiers) andlinks among the entities. For example, with reference to FIG. 2C, thedatabase includes an entity “Rachel.” In light of Rachel's conversationwith Erin, a new person entity “Alan” is added to the database andlinked to “Rachel” via two links (i.e., “fence together,” “servedoff-menu”). Further, a new food entity “Enchiladas del mar” is added tothe database and linked to “Rachel” via a link (i.e., “loves”).Exemplary algorithms can include open-source algorithms (e.g., SpaCy),proprietary algorithms, or any combination thereof.

As another example, for the following utterance “This is a perfectexample of ‘The Adjacent Possible theory’ by Stuart Kauffman where hesuggests at any given moment in history or science, can only makeprogress in certain prescribed ways. For the independence of blindpeople, Aira was possible and bionic eyes (or similar) wasn't yet. I amalways fascinated by the ‘adjacent possibilities’ of hard problems,” theaudio processing system can extract and classify entities such as “TheAdjacent Possible Theory” as a work of art, “Stuart Kauffman” as aperson, and “Aira” as an organization.

In some embodiments, the audio processing system 206 comprisesnatural-language processing techniques for determining intent of thespeaker. In some embodiments, the audio processing system includes oneor more machine-learning models (e.g., neural networks) to determine anintent of the speaker (e.g., based on the transcription such as theverbs, context, history) and identify an action corresponding to theintent. The natural-language processing algorithms can includeopen-source algorithms (e.g., Rasa Open Source), proprietary algorithms,or any combination thereof.

The audio processing system 206 can generate a number of parameters ofthe memory segment. FIG. 2B illustrates the parameters in the memorysegment and the sources from which each parameter can be generated, inaccordance with some embodiments.

With reference to FIG. 2B, the audio processing system can generate atext string 252. In some embodiments, the text string comprises a textstring in the natural-language format. In some embodiments, the textstring has a predetermined maximum length (e.g., 10 words). In thedepicted example in FIG. 1A, the text string can be a text string“Rachel's favorite dish at Las Havas is Enchiladas Del Mar.” In someembodiments, the system generates multiple memory segments havingmultiple natural-language text strings. For example, the system cangenerate a second memory segment “Alan is the chef of Las Havas.” basedon the conversation.

The audio processing system can generate one or more entity parameters254. The entity parameters can be some or all of the entities extractedfrom the audio data by the audio processing system as described above.In the depicted example in FIG. 1A, the entity parameters can include“enchiladas del mar.”

The audio processing system can generate one or more people parameters256. In some embodiments, the people parameters 256 include identifiersof people present at the conversation (e.g., Erin, Rachel). In someembodiments, the people parameters 256 include identifiers of peoplementioned (e.g., Alan) during the conversation. The people parameterscan be derived from the outputs of the speaker diarization algorithmsand/or the natural-language processing algorithms.

The audio processing system can generate one or more date/timeparameters 258. In some embodiments, the date/time parameters 258include the date and/or time when the conversation is captured. In someembodiments, the date/time parameters 258 include the date/timeinformation mentioned during the conversation. The date/time parameterscan be derived from the time stamp(s) associated with the audio data(e.g., time stamp associated with Rachel's speech).

The audio processing system can generate one or more location parameters260. In some embodiments, the location parameters 260 include thelocation of the user when the conversation is captured. The locationparameters can be derived from audio characteristics of the ambientaudio data (e.g., based on known audio signatures associated withdifferent locations). In some embodiments, the location parameters 260include one or more locations mentioned during the conversation.

The audio processing system can generate one or more context parameters262. In the embodiments, the context parameters include conversationtopics or user intents derived from the audio input based onnatural-language processing algorithms. For example, the contextparameters 262 can include information related to Erin's previous visitsto the restaurant.

The audio processing system can generate one or more emotion parameters264. The audio processing system can identify emotions associated with aconversation by detecting audio characteristics (e.g., speech rate,tone, pitch, intonation, energy level) in the speech often associatedwith certain types of emotions. Further, the audio processing system canidentify emotions associated with a conversation by detecting keywords(e.g., “happy,” “thrilled,” “mad”) in the speech transcription oftenassociated with certain types of emotions. Further, the audio processingsystem can identify emotions associated with a conversation bynon-verbal cues (e.g., pauses, sighs, coughs, hesitations) in the speechoften associated with certain types of emotions.

In some embodiments, the emotion parameters include emotion parametersdirected to in-memory emotion/sentiment, emotion parameters directed toreaction/attitude to the memory. In-memory emotion/sentiment compriseemotions extracted by the system at the time of memory creation. Incontrast, reaction/attitude comprise attitude toward the memory at anyother time (i.e. “liking” a memory or otherwise reacting with emotion,or programmatically extracting attitude toward the memory at any pointother than when it is being created, such as worries before an event orpositive feelings about an event after the fact).

The audio processing system can generate one or more owner parameters268. In some embodiments, the owner of a memory segment is the user ofthe memory retention system (e.g., Erin). In some embodiments, the ownerof a memory segment is the speaker of the audio input (e.g., Rachel).

Turning back to FIG. 2A, the memory retention system can include alocation processing system 208 for processing location data in thememory data 102. As shown in FIG. 2B, the location processing system 208can generate one or more location parameters 260 of the memory segment.For example, the memory data 102 can include GPS coordinates of Erin'swearable device (e.g., glasses 102) when Erin met Rachel at therestaurant. Based on the GPS coordinates, the location processing system108 can identify an address and the associated business entity (e.g.,“Las Havas”), which can be included as location parameters 260 in thememory segment.

The location processing system 208 can generate one or more contextparameters 262. For example, the system can, based on the location data,derive when Erin previously went to that restaurant, who Erin previouslymet at that restaurant, and/or other related memories of Erin's at thatrestaurant. The derived information can be included as contextparameters.

The location processing system 208 can generate one or more visualmemory parameters 266. For example, based on the GPS coordinates, thelocation processing system 108 can identify an address and theassociated business entity (e.g., “Las Havas”). Based on the address andthe business entity, the location processing system can obtain images ofthe address or the business entity from a third-party database andinclude the images as visual memory parameters 266 in the memorysegment.

Turning back to FIG. 2A, the memory retention system can include animage processing system 209 for processing image data in the memory data202. As shown in FIG. 2B, the image processing system 209 can generateone or more people parameters 256, date/time parameters 258, locationparameters 260, and emotion parameters 264 of the memory segment. Forexample, the memory data 202 can include a photo taken by Erin'swearable device (e.g., glasses 102) or Erin's mobile device at therestaurant. The image processing system 209 can use facial recognitionalgorithms to identify people (e.g., Rachel, Erin) and their emotions inthe photo. The photo include metadata such a time stamp and a location,which can be included as parameters in the memory segment. Further, thephoto and a portion thereof can be included as visual memory parametersin the memory segment.

Turning back to FIG. 2A, the memory retention system can include asocial media processing system 210 for processing social media data inthe memory data 202. As shown in FIG. 2B, the social media processingsystem 210 can generate one or more entity parameters 254, peopleparameters 256, date/time parameters 258, location parameters 260,emotion parameters 264, and visual memory parameters 266 of the memorysegment. For example, the memory data 202 can include a post made byErin (e.g., “Having dinner with @Rachel at @LasHavas restaurant. Thankyou @Alan:)”) on social media. The social media processing system 210can thus extract various information such as entities, people, emotions,time, and location from social media posts. In some embodiments, asocial media post can include one or more photos and videos, which canbe processed by the image processing system 109 to generate additionalparameters of the memory segment and included as visual memoryparameters.

Because portions of memory data 202 (e.g., audio data, location data,image data, social media data) are associated with time information, theaudio processing system 206, the location processing system 208, theimage processing system 209, and the social media processing system 210can generate parameters corresponding to a same time period and groupthese parameters in a single memory segment. In some embodiments, thememory retention system can de-duplicate and merge parameters generatedby the various processing systems.

In the depicted example, a memory segment is generated based on Rachel'sutterance. It should be appreciated, however, that a memory segment canbe generated based on a part of an utterance, multiple utterances, or aconversation.

Turning back to FIG. 2A, at block 214, the memory retention systemstores the memory segment. In some embodiments, the memory retentionsystem comprises multiple databases including a time-series database anda graph-based database (e.g., FIG. 2C). The time-series database allowssearching of memory segments by time. In some embodiments, thetime-series database includes a plurality of memory segments (orportions thereof) and indexes the plurality of memory segments with timestamps.

The graph-based database allows searching of memory segments bysimilarity. For example, the memory retention system can add nodes andrelationships to the graph-based database as described with reference toFIG. 2C. In some embodiments, the memory segment or a portion thereof(e.g., the text string) is stored in association with the nodes and/orrelationships in the graph.

In some embodiments, the memory retention system stores the memorysegment using a blockchain-based platform (e.g., Ethereum, Oasis Labs).In some embodiments, each memory segment is stored as a block in ablockchain-based database. Each memory segment can be attached to a nonfungible token (NFTs) for memory exchange between users.

The process 200 can be performed on a single device (e.g., a wearabledevice) or multiple devices (e.g., a client-server system). For example,each of the systems 206-210 can be implemented by one or multipledevices.

FIG. 3A illustrates an exemplary process 300 for retrieving a memorysegment or a memory segment portion, in accordance with someembodiments. Process 300 is performed, for example, using one or moreelectronic devices implementing a software platform. In some examples,process 300 is performed using a client-server system, and the blocks ofprocess 300 are divided up in any manner between the server and a clientdevice. In other examples, the blocks of process 300 are divided upbetween the server and multiple client devices. Thus, while portions ofprocess 300 are described herein as being performed by particulardevices of a client-server system, it will be appreciated that process300 is not so limited. In other examples, process 300 is performed usingonly a client device (e.g., wearable device 102) or only multiple clientdevices. In process 300, some blocks are, optionally, combined, theorder of some blocks is, optionally, changed, and some blocks are,optionally, omitted. In some examples, additional steps may be performedin combination with the process 300. Accordingly, the operations asillustrated (and described in greater detail below) are exemplary bynature and, as such, should not be viewed as limiting.

With reference to FIG. 3A, an exemplary system (e.g., one or moreelectronic devices) receives one or more user inputs 302 indicating aquery for a memory segment portion. The system can allow a user tosubmit the query via a plurality of input devices. For example, thesystem can include a wearable device (e.g., glasses 102) that canreceive auditory inputs (e.g., a natural-language utterance such as“what was the name of the dish Rachel mentioned last night”). As anotherexample, the system can include a wearable device (e.g., glasses 102)that can detect facial expressions or gestures as described withreference to FIGS. 4A-C.

As another example, the system can include a software application orservice (e.g., a chat bot) that can receive textual inputs (e.g., anatural-language text string such as “Rachel's favorite dish” or “dishRachel mentioned last night”) via an electronic device such as a mobilephone. As yet another example, the system can provide functionalitiesintegrated with a third-party device or service (e.g., a third-partydigital assistant) that can receive and process auditory inputs (e.g.,“Alexa, what's the dish Rachel mentioned last night?”).

At block 304, the system formulates a query for a memory segmentportion. For example, if the user input 302 is a natural-language textstring (e.g., “dish Rachel mentioned last night”), the system can useone or more natural-language algorithms to determine a user intent(e.g., “search for food item”) and search parameters (e.g., “Rachel,”“mentioned,” “last night”).

In some embodiments, at block 305, the system determines whether a queryfor memory segment portion has been made. For example, a specificsequence and/or combination of user inputs can indicate that a query formemory segment portion has been made. For example, the system candetermine that a query for a memory segment portion is made if the userinputs 302 comprises an eye blink exceeding a predefined duration (e.g.,1 second).

In some embodiments, the system can comprise an action processing system308 for processing facial expressions of the user, as described withrespect to FIGS. 4A-C. For example, the action processing system 308determines whether the duration of the eye blink exceeds a predefinedduration (e.g., 1 second). In some embodiments, if the eye blink exceedsthe predefined duration, the system obtains and processes an audiosegment preceding the eye blink (e.g., a 10-second audio segment, a15-second audio segment) to formulate the query using the audioprocessing system 306.

In some embodiments, the system continuously or periodically processesan audio segment from (t−10) seconds to t (e.g., t represents thepresent). If the system detects an eye blink exceeding a predefinedduration, the system can complete formulating the query based on anaudio segment preceding the eye blink. Because the processing of theaudio segment has been initiated before the system detects the eyeblink, the time the user needs to wait to retrieve memory data afterblinking can be significantly reduced.

In some embodiments, the system can comprise an audio processing system306 for processing auditory inputs to formulate a query. The audioprocessing system 306 can be the same as, or share components with, theaudio processing system 206 (FIG. 2A). The audio processing systemcomprises natural-language processing algorithms to determine intent ofthe user (e.g., searching for a person, searching for a place) andformulate the query (e.g., identifying the search parameters).

In some embodiments, the audio processing system 306 comprises adeterministic intent parser (e.g., regular expressions) and aprobabilistic intent parser (e.g., logistic regression). The audioprocessing system 306 can use the deterministic intent parser and thenuses the probabilistic intent parser if the output of the deterministicintent parser does not exceed a confidence threshold.

The deterministic parser relies on regular expressions to match intentand slots. The probabilistic parser relies on machine learning togeneralize beyond the set of sentences seen at train time. This parserinvolves two successive steps: intent classification and slot filling.The intent classification step relies on a logistic regression toidentify the intent expressed by the user. For example, for an utterance“Remind me who formulated the adjacent possible theory,” theprobabilistic intent parser can formulate a structured query such as:

{

“intent”: “search_person”,

“entities”: {

“work_of_art”: “adjacent possible theory”,

}}

In some embodiments, the system can determine that a query for a memorysegment portion has been made if the user inputs 302 comprises a speechinput having a query intent (e.g., “what was the name of the dish Rachelmentioned last night?”) even if the speech input is not followed by aneye blink. For example, the system can continuously receive audio data(e.g., from a microphone of a wearable device) and determine the user'sintent (e.g., whether the user utterance is directed to the system or toothers). The system can determine whether the user utterance is directedto the system, for example, by determining that the user is not engagedwith a conversation with others.

At block 310, the system retrieves the memory segment portion based onthe formulated query. In some embodiments, at block 312, the systemselects a database based on the formulated query. If the query is basedon similarity (e.g., “what is Rachel's favorite dish”), the systemselects the similarity-based database (e.g., graph-based database). Forexample, the system identifies, in the graph-based database, the nodecorresponding to “Rachel,” and identifies an associated nodecorresponding to “favorite food.”

If the query is based on time (e.g., “what dish did Rachel mention lastnight?”), the system selects the time-series database. For example, thesystem identifies a memory segment that has a time stamp matching “lastnight” and a people parameter matching “Rachel.” The system can thenidentifies the text string in the memory segment “Rachel's favorite dishas Las Havas is enchiladas del mar.”

At block 314, the system outputs the retrieved memory segment portion.In the depicted example in FIG. 1A, the system outputs the retrievedmemory segment portion by displaying the text string “Rachel's favoritedish as Las Havas is enchiladas del mar” via glasses 102. In thedepicted examples in FIG. 1B, the system can output the entire memorysegment on a mobile device 104. As shown, the mobile device 104 displaysthe text string (referred to as “Membit 106”), as well as the associateddate/time parameters 108, location parameter 110, social mediaparameters 112, emotion parameters 114.

As depicted in FIG. 1C, the system can output the retrieved memorysegment portion via a notification on a smart watch 120. As depicted inFIG. 1D, the system can output the retrieved memory segment portion viaan audio output on a speaker 130. In some embodiments, the speaker 130includes a built-in third-party digital assistant, which can beintegrated with the system to process user queries and retrieve memorysegments.

As depicted in FIG. 3B, the glasses can provide a textual output (“Theplay is at San Diego Civic Center in downtown”) and a correspondingaudio output in response to an utterance (“Where is Grace's play,again”). As depicted in FIG. 3C, the glasses can automatically derive asecond user intent from the utterance (i.e., navigation) in addition tothe intent to query the system and perform a follow-up task (e.g.,providing navigation information).

In some embodiments, the memory retention system provides a timeline ofa user's memory via one or more user interfaces. FIG. 3D illustrates anexemplary timeline user interface displaying the user's memory segmentsin a timeline format, in accordance with some embodiments. The timelineuser interface can be accessed via one or more electronic devices (e.g.,a mobile app, a software application, a software service, a wearabledevice). In the depicted example, at the “Now” point on the timeline, atranscript of the current conversation is displayed. At the top of theuser interface, the most recent or the current memory segment formulatedcan be indicated.

FIGS. 4A-C illustrate exemplary electrooculography techniques of amemory retention system, in accordance with some embodiments.Electrooculography refers to measurement of the electrical potentialbetween electrodes placed at points close to the eye to detect eyemovements. A human eye can be modeled as a dipole with its positive poleat the cornea and its negative pole at the retina. Assuming a stablecorneo-retinal potential difference, the eye is the origin of a steadyelectric potential field. The electrical signal that can be measuredfrom this field is the electrooculogram (“EOG”). EOG is used to detecteye movements, including blinks, fixations (i.e., eye's stationary stateduring which gaze is held), saccades (i.e., constant eye movementbuilding a mental map of a scene). Other movements (e.g., repetitivemovement due to reading) can be detected using a combination of thethree mentioned. Electrooculography generally is less computationallyintensive and requires less power than tracking eye movement usinghigh-resolution cameras.

With reference to FIG. 4A, five electrodes can be positioned to detectelectrical signals due to eye movements: a pair of vertical (V),Horizontal (H) electrodes and a reference (R) electrode. Theseelectrodes can be affixed to a wearable device (e.g., glasses 102) tomaintain their relative positions. The complexity of the electrodearchitecture may depend on number and type of eye movements that areneeded to be detected.

For detecting blinks only, H electrodes may be eliminated in someembodiments. As shown in FIGS. 4B-C, the circuits and Reference (R)electrode are on the side and the Vertical (V) electrodes are above andbelow the eye.

In some embodiments, the audio inputs received by the memory retentionsystem can be transmitted to a server device for processing. The voicetransmission can be performed via VoLTE or Over the Top (OTT) VoIP, insome embodiments.

FIG. 6A illustrates exemplary data structures for storing memory data,in accordance with some embodiments. The memory retention systemsupports a plurality of types of data structures: memory bit 602, memorysnippet 604, memory block (or memory event) 606, memory chunk 608, andmemory stack 610. In some embodiments, each user of the memory retentionsystem is associated with his or her own memory stack 610.

A memory bit can refer to a basic information unit such as a location, aperson, a time, etc. A memory snippet can refer to an utterance such asan audio sentence or a textual sentence (e.g., a single Slack message).A memory block can refer to a plurality of consecutive utterances fromthe same person (e.g., an email) or multiple people. A memory chunk canrefer to a grouping of utterances that involve multiple people. A memorychunk or memory event can refer to a single conversation, an event(e.g., calendar event), one voice event, one email thread, etc.

The data structures can be configured to store content data (e.g.,emails, tweets, audio transcriptions), and context data or contextparameters, which includes metadata (e.g., people, time, location,source) and generated data (e.g., emotions, summaries) as describedherein. All of the memory data structures described herein can be storedin a blockchain-based database.

The various data structures are in parent-child relationships. Forexample, a memory bit is child to a parent memory snippet; a memorysnippet is child to a parent memory block; a memory block is child to aparent memory chunk. For example, in the exemplary data segment in FIG.2B, the audio utterance can be stored as a memory snippet, while theparameters can be stored as children memory bits to the memory snippet.

Each type of data structures 602-610 can include metadata and generateddata in addition to content data (e.g., text, audio, image, video). Forexample, a memory block can include content data (e.g., transcript ofutterances), metadata (e.g., location where the utterances are spoken,the speaker) and generated data (e.g., an emotion associated with theutterances). As another example, a memory chunk can include, asgenerated data, a summary of all of the utterances in the child memoryblocks.

FIG. 6B illustrates an exemplary memory retention system 650 forgenerating memory data structures and retrieving memory data structures,in accordance with some embodiments. The memory retention system 650 canbe used to perform the method of FIG. 2A.

With reference to FIG. 6B, the memory retention system comprises aplurality of feed modules 652 for receiving memory feed data (e.g.,memory data 202). The feed modules can include a memory retentionapplication 654 and one or more third-party applications 656. The memoryretention application 654 can include a standalone application or acomponent (e.g., plug-in) of another application. The memory retentionapplication can operate on any type of electronic devices, such asmobile devices, laptop devices, audio devices, vehicles, and wearabledevices (e.g., devices in FIG. 1B). As shown in FIG. 6B, the memoryretention application can be configured to receive speech data 660 a,text data 660 b, and media data 660 c.

The third-party applications 656 can comprise any type of softwareapplications that receive user data, including conferencing applications(e.g., WebEx, Teams, Zoom), social media applications (e.g., Twitter,Facebook, Instagram), media applications (e.g., YouTube, Google Photos),collaboration applications (e.g., Slack), productivity applications(e.g., Google Calendar, Outlook), etc. These applications can beintegrated into the memory retention system to provide speech data 660a, text data 660 b, and media data 660 c.

The speech data 660 a (e.g., utterances during a Zoom call) can beprocessed by an audio processing system 662 (e.g., audio processingsystem 206). As discussed above (e.g., FIG. 2B), the audio processingsystem processes the speech data to obtain content and context data. Forexample, the audio processing system can comprise speech recognitionalgorithms to obtain a text transcription from the speech data 660(i.e., content data). In some embodiments, user-specific speechrecognition models can be used to enhance speech recognition. The audioprocessing system can further comprise speaker diarization algorithms toassign speaker IDs to various portions of the text transcription. Insome embodiments, the text transcription is further processed by thetext processing system 664 to segment the transcription into a pluralityof segments (e.g., sentences).

The text data 660 b (e.g., emails, text messages) can be processed by atext processing system 664. As discussed above (e.g., FIG. 2B), the textprocessing system 664 can process the text data to obtain content andcontext data. For example, the text processing system can perform textsegmentation to segment the text data 660 b into a plurality of segments(e.g., sentences). Further, speaker IDs can be assigned to the segmentsbased on the metadata associated with the text data (e.g., author of themessage or email).

The media data 660 c includes image and video data and can be processedby a media processing system 668 (e.g., image processing system 209). Asdiscussed above (e.g., FIG. 2B), the media processing system can processthe media data to obtain content and context data. For example, themedia processing system can recognize objects and entities in the mediadata.

With reference to FIG. 6B, a snippet transformer 670 generates aplurality of memory snippets. A memory snippet can include asingle-sentence utterance (e.g., “Do you have a dog in your house rightnow?”) or a single text message (e.g., “My vision for Luther AI is tocreate a technological version of human memory that is available for theuser to recall any information they want”). A memory snippet can havechild memory bits (e.g., entity, time, location) that can be generatedbased on the memory data as described herein.

Further, the memory retention system can generate memory blocks 674based on the memory snippets 672. A memory block can refer to aplurality of consecutive utterances from the same person (e.g., anemail, a monologue). A memory block can have a plurality of child memorysnippets. For example, a memory block can be generated based on aTwitter thread, with tweets from a person stored as child memorysnippets of the memory block. Metadata (e.g., time, people, location)can be derived from the third-party platforms (e.g., Twitter, Gmail) andassociated with the memory block and/or child memory snippets.

Further, the memory retention system can generate memory chunks based onmemory blocks. For example, the system can automatically group memoryblocks 674 into the same memory chunk or memory event. This grouping canbe made if the utterances are close to each other in time, if theutterances are associated with the same location, and/or if theutterances are associated with the same calendar event.

For example, from a calendar meeting in a user's calendar, a memoryevent can be generated. Metadata associated with the calendar meeting(e.g., time, duration, location, people, source) can be derived from thecalendar application and associated with the generated memory event.Further, audio utterances obtained during the time of the meeting by thememory retention system can be stored as child memory blocks and memorysnippets. As another example, a memory chuck can be generated based onan email thread, with each email stored as a child memory block.

With reference to FIG. 6B, a plurality of transformers can be used toaugment the memory data structures (e.g., snippets, blocks, chunks). Theoutputs of the transformers can be stored as new memory data structures(e.g., synthetic data structures) or as generated data associated withexisting memory data structures. The Stack transformers includesartificial intelligence (“AI”) models that extract and predictattributes to enrich raw memory blocks—applied to the user's memorystack at runtime when the user takes the action to feed. The Recalltransformers includes AI models that retrieve and generate answers fromrelevant memory data—retrieved from the users' memory stack at runtimewhen the user takes the action to recall.

In some embodiments, the transformers include a transformer 676 a forextracting entities. As described above, the transformer can process anutterance (e.g., “Do you have a dog in your house right now”) in amemory snippet to identify one or more entities (e.g., “DJ sister'sdog”). The extracted entities can be stored as child memory bits of thememory snippet.

In some embodiments, the transformers include a transformer 676 b foridentifying concept trends. From the extracted entities from each of thememory blocks, the system can compute the concept trends by identifyingthe most significant terms based on context—recency (time), event(current), and people (context) and surface the top concepts that aremost relevant (e.g., most recent, most frequent, most discussed with aparticular person).

In some embodiments, the transformers include a transformer 676 c foridentifying related memories. Related memories are generated frommultiple sources. Related memories are represented as memory blocks thatare related: the related-ness between memory blocks is determined basedon a measure of similarity between the context (e.g. metadata andgenerated data) computed from graph nodes and based on a measure ofsimilarity between the content (e.g. text, image, audio) computed fromraw and transformed data.

In some embodiments, the transformers include a transformer 676 d foridentifying emotions. For example, the transformer can identify emotionsby detecting audio characteristics in the speech, keywords andpunctuations, and non-verbal cues, as described herein.

In some embodiments, the transformers include a transformer 676 e forparaphrasing. The paraphrasing transformer creates a grammar andsemantic corrected version of the memory block. Language models areapplied to both transcribed text and the written text without modifyingthe original meaning.

In some embodiments, the transformers include a transformer 676 f foridentifying insights. The insight transformer serves user analytics andtrends (e.g. numeric or qualitative) on metadata and transformerpredicted/generated values. Examples include the majority of the emotionlast week, the number of facts identified, etc.

In some embodiments, the transformers include a transformer 676 g foridentifying dialog act. The dialog act transformer identifies thefunction of the sentence such as greetings, questions, statements,facts, opinions, acceptance. This is based on categorizing the type ofthe sentences in a conversational setting.

In some embodiments, the transformers include a transformer 676 h forquestion generation. The question generation transformer generatesquestion answer pairs given the context of a memory block or memorysnippet. The question generation identifies the facts in the memoryblock and the question context based on the fact the user can recalllater.

In some embodiments, the transformers include a transformer 676 i forproviding summarization. The summarization transformer is based ongeneration both extractive and abstractive summary based on both theindividual memory data structures and multiple memory data structures.For example, an extractive summary of a memory block is one or twosentences that contains the most relevant concepts for the user. Anabstractive summary of multiple memory blocks contain generatedsentences for multiple relevant concepts for the user.

With reference to FIG. 6B, the memory retention system further comprisesa plurality of recall transformers. The entity linking transformer is acontext based recall transformer based on metadata and transformedvalues (e.g. memory bits) by internally linking to other memory blocksthat share similar values or externally linking to other content (e.g.Wikipedia) by similar values. The QA retrieval transformer is a factbased recall transformer based on open domain question answering byextracting the answer to a factual question given a context of one ormore memory blocks. The generative transformer (predict next) is agenerative based retrieval transformer based on generating possibleanswers given a context of a few words or sentences, auto completing asingle or multiple sentences. The recall transformers can be invokedwithin the memory retention app 654 and within third-party platforms(e.g., Twitter). Further, it can be invoked to process query terms asthey are provided in real time (e.g., as the user inputs a query in theuser interface 700) or process existing query terms (e.g., to processtext in an existing Google Docs document).

The memory data structures allow full-text and full-element searchesacross different levels. In some embodiments, the memory data structurecan be in a graph model, with context data (metadata and generated data)as memory nodes and the memory data structure (e.g., memory blocks) asmemory edges. For example, nodes can be “Suman”, “February 2020”,“patent application” with edges between them as “I was working on apatent application for Luther Labs” with timestamp 20200228 and “I amworking on a patent application for Human AI Labs” with timestamp20210224. The temporal information kept on the edges enables the systemto create a time series representation of all the historical memoryblocks between two memory nodes and the node representation in theembedding space leverages both the time information and the free textinformation on the edges.

Any of the transformers described herein can include an AI ormachine-learning model to improve its performance over time. The modelcan be trained based on the user's memory stack, bootstrapped data, orany combination thereof. In some embodiments, the transformers cangenerate synthesized memory data structures. These memory structures canbe generated using one or more user-specific models (e.g., GPT stylelanguage model).

For example, the system can generate a model about a specific person(e.g., based on history of all communications with that person) toanswer recall queries regarding the person (e.g., “How many times have Imet Rachel”). As another example, the system can generate a model abouta specific time (e.g., based on an event, a week, a day) to answerrecall queries regarding the time (e.g., “What did I do last week?”“When was the last time I attend this conference?”) In other words, thesystem can generate a model about a specific context (e.g., based on asingle concept) to answer recall queries regarding the context (e.g.,“My tweets about my vision for this company”).

In some embodiments, the models are trained to conform to the user'sspecific style for writing text and/or speaking (e.g., providing outputsbased on the user's specific style). For example, the model generatesdifferent styles of the same semantic content in the short form ofTwitter (e.g. “My vision for Luther is to create a technology version ofhuman memory. #vision #ai) and E-mail (e.g. “A little about our company:our vision for Luther is to create a technological version of memory toaugment the human biological capacity.”). These models can be eitherunsupervised models or supervised models. In some embodiments,supervised modeling is restricted to the use cases that does notintroduce bias—such as question answering.

In some embodiments, user feedback (implicit and explicit) can be usedto measure model efficacies and train/improve the models. In someembodiments, the user can explicitly provide memory data to the systemto retain. For example, the user can speak to the system (e.g., “from1-2 pm I talked to Will about Documentation”) to create new memory datastructures or provide metadata about existing memory structures. Theuser can add a list of related general topics that the system can modelto automatically tag other memory blocks on a similar topic. The contentof each memory block is compared and classified against availableknowledge sources and common topics that occur in them. (e.g. Wikipedia)

In some embodiments, the user can edit or add tags into the system(e.g., via user interface 800). The tags are in turn incorporated intothe corresponding memory data structures, the speech recognition models,transformer models, and/or the user graph (as memory nodes, memoryedges, etc.). The user can correct transformer outputs, such asemotions, dialog act, speaker, and other information that may be usedfor recall and training purposes. In some embodiments, through useractions (e.g. accept, shuffle) on the recall user interfaces (e.g., userinterface 700), the system can construct or update user graphs from therecalling behavior and dynamically adjust to predict what the user maywant to know. (e.g. what is likely to be recalled together).

With reference to FIG. 6B, the memory retention system comprises aplurality of recall modules. The recall modules can include the memoryretention app 654. Exemplary user interfaces of the memory retention appare described herein with respect to FIGS. 1A, 1B, 3B-D, 7A-C, and 8A-C.The recall modules can include third-party applications, includingconferencing applications (e.g., Slack, Teams), text applications (e.g.,SMS, Whatsapp), virtual assistants (e.g., Alexa, Siri), and social mediaapplications (e.g., LinkedIn, Twitter). The third-party applications canprovide recall functionalities using native user interfaces. Forexample, a user may ask Siri “Hey Siri, what was the name of the dishRachel mentioned last night?” As another example, a user may enter aquery for memory in the search bar of Slack. In some embodiments, entrypoints to the memory retention application 654 can be provided in thethird-party applications. Exemplary user interfaces are described hereinwith reference to FIGS. 9 and 10.

FIGS. 7A-C illustrate exemplary user interfaces of a memory retentionsystem, in accordance with some embodiments. The memory retention systemprovides user interfaces that are easily accessed and operated,providing a seamless and intuitive experience that does not require muchtraining, explanation of documentation.

In some embodiments, the user interfaces in FIGS. 7A-C can be invokedwithin a memory retention application (e.g., memory retentionapplication 654) implemented on an electronic device (e.g., mobiledevices, audio devices, vehicles, wearable devices, computers). In someembodiments, the user interfaces can be invoked within third-partyapplications (e.g., third-party applications 690) or within an operatingsystem.

In some embodiments, the user interfaces can be invoked using auniversal hotkey (e.g., command+R). Responsive to receiving activationof the hot key, the user interface 700 is displayed for recalling memorydata. When invoked, the user interface 700 may include previous results,which disappear as soon as the user starts speaking or typing. In someembodiments, the user can press a predefined key (e.g., <ESCAPE>) todismiss the user interface 700.

With reference to FIG. 7A, the user interface 700 comprises a text box704 for entering a recall query. The user can enter the recall query bytyping into the text box or providing an audio input. The query can beformulated as either a statement (“My vision is”) or a question (“What'sRachel's favorite restaurant?”).

With reference to FIG. 7B, as the user enters the recall query, one ormore auto-complete options (e.g., for people, location, time, concepts)are provided. For example, the system provides the one or moreauto-complete options once the confidence measure for the predictionexceeds a predefined threshold. The options can include additional queryterms. In the depicted example, in response to the user input “My visionfor,” the system provides a plurality of options: “Human Ai,” “Aka,” and“Privacy.” The user can scroll through the options by pressing a firstpredefined key (e.g., <SHIFT>) and select an option by pressing a secondpredefined key (e.g., <TAB>).

In some embodiments, the ranking of these options can be dependent ontheir relative relevance, user-specific preferences and history (e.g.,user favorited, recently searched, recently tagged), and/or top-of-mindpeople/topic (e.g. next events in calendar, significant concept tags).

The list auto-complete options can be updated as the user provides moreterms to complete the query. With reference to FIG. 7C, after the userselects “Human Ai,” the system can display a new auto-complete option.The user can press a third predefined key (e.g., <ENTER>) to display thespecific memory data structure (e.g., memory snippet, memory block)within his or her memory stack. Exemplary user interfaces for such deeprecall are described below with reference to FIGS. 8A-C.

In some embodiments, when the user interface 700 is invoked, the textbox 704 can provide one or more autocomplete options before any userinput is provided, such as: “Does,” “What,” “Why,” “How,” “I met with,”“I wrote to,” “I asked,” “I felt,” “thetime/person/place/reason/thing/amount/feeling/thought/concept/topicthat,” etc. In some embodiments, if there is existing context in thebuffer (e.g., the user has spoken before activation of the recall userinterface), the existing context forms the beginning of the query.

In some embodiments, the user interface 700 provides entry points forobtaining additional information and/or invoking additional userinterfaces, such as the last memory block recalled, the timeline userinterface in FIG. 8A, a recent recall query, the last topic discussed,the last question asked, the last thing the memory retention systemlearned (e.g., “Rachel's dog is called Moe”), Favorite Memory, etc. Someor all of the information or user interfaces may be invoked usinghotkeys (e.g., command+a predefined key).

FIG. 7D illustrates another exemplary user interface, in accordance withsome embodiments. Responsive to a user query, a drop-down menu 708 isdisplayed to provide a summary of the results from the query. In thedepicted example, the drop-down menu indicates how many auto-completesuggestions are generated, how many top memory constructs (e.g., memoryblocks) are identified, and how many relevant memory data structures areidentified. The user can press <shift> to cause the auto-completesuggestions to be displayed one by one in text box 704. The user canpress <enter> to launch a user interface for displaying the top memoryconstructs, as shown in FIG. 7E. As shown in FIG. 7E, the top memoryconstructs can be displayed in one or more batches and the memoryconstructs in each batch are displayed in an arrangement for easyexploration. The user can also invoke the memory stack from thedrop-down menu to launch a timeline user interface (e.g., 8A) to viewand explore the full set of relevant memory data.

The recall queries for the memory retention system can be specific,detailed, and broad. A specific query seeks a single entity (e.g.,person, place, thing, time, concept), such as “Luther's CEO is.” Adetailed query seeks a longer answer composed of several linkedentities, such as “who did I meet at Mac's Pub,” “who did I visit withMarc in San Francisco,” “who did I meet as a child,” “who did I meetduring college.” A broad query seeks a list of several unlinkedentities, such as “what did I do last week.”

The categories of information that can be recalled or used as queryterms include: metadata (e.g., person, time, location), content data(e.g., audio transcripts, texts, images), and generated data (e.g.,emotion, summary). Examples include: person, location (specific locationsuch as will's house and broad location such as Colorado), time(specific time such as an exact date/time, or a time range), an event(e.g., honeymoon, wedding, graduation, party), life stage (child,college student, married), a concept, a complete thought (e.g., asnippet), synthesized thought (e.g., a summary based on multipleblocks), etc. For example, the user may ask the system to “generate amemory timeline synopsis around a concept or topic we talk about.”

In some embodiments, the recall queries are metadata-driven, forexample: “When was the last time I met Rachel?” or “What did we talkabout last time I emailed Rachel?” Metadata refers to attributes (e.g.,people, time, location) that can be derived from feed sources (e.g.,audio, calendar applications, email applications, messagingapplications, sensors) or obtained from user feedback (e.g., tagging).In some embodiments, metadata further indicates the feed source of thememory data (e.g., the third-party application the data is from) and thefeed type of the memory data (e.g., audio, text, image). Metadatainformation is explicit and accurate. The memory retention system canstore metadata as memory bits, and each memory bit can be associatedwith a memory snippet, a memory block, and/or a memory event.

Metadata-driven queries are often formulated to contain <recall verb><person> <time> < . . . >. The system can extract the recall verb fromthe query and derive query terms such as feed source, feed type,context, and emotion based on the recall verb. For example, [I met with]indicates that the source may be a calendar application, [I e-mailed]indicates that the source may be an email application, [I wrote to]indicates that the source may be a text application, [I spoke with]indicates that the source may be an audio application, [I asked]indicates that the memory is associated with a question, [I laughedwith] indicates that the memory is associated with an emotion, etc.

The time in the query can be classified as relative time (specifying therelation between capture time and the current time such as “last time”),absolute time (e.g., “first time,” “this Tuesday,” “June 2020”),internal time, or event time. Further, the time can have differentresolutions (e.g., month, week).

In some embodiments, the recall queries can driven by content data(e.g., “What did I say to Rachel at 2 pm yesterday”) and/or by generateddata (e.g., “What emotion did I feel during my call with Rachel”). Basedon the query terms, the corresponding memory data structures can beretrieved as described herein.

In response to receiving a query, the memory retention system canproduce results for the user. The results can include the precise answer(e.g. a memory bit), a sentence (e.g. memory snippet), a paragraph (e.g.memory block), and/or media data (e.g., images and videos). For example,a memory bit (e.g., time, people, location) is generally retrieved tocomplete a sentence, such as “I spoke to Kelly last time on <yesterday>”and “I spoke to Lakshmi about Diwali on <Saturday>.” A memory snippet ora memory block is generally retrieved when the user wants to recall oneor more sentences. For example, based on the user query “What did I tellWill about AWS cost on Monday,” the system can retrieve one or morememory snippets/blocks associated with the concept or conversation.

In some embodiments, the system can automatically suggest changes to theuser query. For example, if the system is not able to identify anyresult in response to the query, the system may suggest removing somequery terms such that results can be returned. As another example, ifthe system identifies too many results (e.g., exceeding a predefinedthreshold), the system can automatically offer additional query terms tonarrow the query.

In some embodiments, the system provides a timeline user interface forthe user to view and explore his or her memory stack. The timeline userinterface can be invoked, for example, when a user query (e.g., in userinterface 700) returns too many results (e.g., exceeding a predefinedthreshold). In some embodiments, the timeline user interface can beinvoked when the user wants to explore a particular memory datastructure. For example, FIG. 7C, the user can invoke deep recall toreview the recalled memory data in the context of the timeline userinterface. In some embodiments, the system provides an entry point tothe timeline user interface from the user interface 700.

FIG. 8A illustrates an exemplary user interface for exploring a user'smemory stack, in accordance with some embodiments. With reference toFIG. 8A, in a panel 802, a plurality of memory segments are displayed.In the depicted example, a memory segment may correspond to a memoryblock described with reference to FIGS. 6A and 6B.

An exemplary memory segment 804 includes the content 806 (e.g., an audiotranscription, an email, a tweet, a message, a video). The memorysegment 804 further includes various context information associated withthe memory segment. As discussed above, context information can includemetadata and generated data, such as a graphical element 708 indicativeof the author of the memory segment (e.g., the person who spoke theutterance), graphical elements 810 indicative of people associated withthe memory segment (e.g., participants of the same meeting orconversation), and a graphical element 812 indicative of the source ofthe memory segment (e.g., Zoom, Slack, Twitter, Facebook). The memorysegment 804 also includes an indicator 814 of an emotion associated withthe memory segment and an indicator 816 of a synopsis of the content.The memory segment 804 also includes one or more tags 820. Tags 820include metadata and generated data associated with the memory segment.The data can be automatically generated by the system, provided by theuser, or any combination thereof. The tags can be added, removed, ormodified by the user. For example, the system may automatically derivemetadata for a memory segment based on a third-party calendarapplication (e.g., time, people, location) and generate data (e.g.,emotion, summary) based on a transformer. The metadata and the generateddata for the memory data structure can be displayed as tags in thetimeline; the user can then modify, remote, or add tags as needed.

The user interface allows the user to manage the memory segments in hisor her memory stack. For example, a user can favorite one or more memorysegments (822), share or keep private one or more memory segments (818),archive one or more memory segments, permanently delete one or morememory segments, and/or export one or more memory segments to adifferent format (e.g., Excel, PDF, or JPG). After a memory segment isarchived, it will be removed from the user's Stack and only visible inan archive section.

The user interface comprises a timeline 830. The zoom level of thetimeline 830 can be adjusted by user control 840 to one of a pluralityof predefined levels (e.g., years, months, weeks, days, hours). Thetimeline 830 can act as a scrollbar. When a user is scrolling up anddown his or her feed panel 802, the timeline 830 can be updated toindicate the memory segments currently displayed in the panel 802.

The timeline comprises a plurality of clickable dots. Each dot canrepresent a single memory segment or a collection of memory segments.The size of the dot is indicative of the number of memory segments inthe corresponding collection. In some embodiments, hovering on a dot cancause a tooltip with the number of memory segments to be displayed. Theuser can select a dot to select the corresponding memory segment orcollection of memory segments on the timeline. Upon a user selection ofa dot, the selected memory segment(s), as well as neighboring memorysegments, can be displayed in the panel 802.

In some embodiments, the user can drag a time travel marker 832 alongthe timeline 830. In some embodiments, based on the scrolling speed(e.g., speed of dragging or mouse scrolling), the zoom level of thetimeline can be updated accordingly. For example, as the scrolling speedof the time travel marker 832 increases, the timeline shifts from weeksto months to years.

With reference to FIG. 8B, the timeline comprises a user interfaceelement 840 for selecting a predefined time point (e.g., most recent,yesterday, last week, last month, last year) or specifying a particulartime. Upon a selection of the time, the timeline and the panel areupdated accordingly.

The timeline provides a number of mechanisms for a user to select memorysegment(s). A single memory segment can be selected from the timeline(e.g., 832 in FIG. 8A) or from the feed panel (e.g., 850 in FIG. 8C).Further, the user can add multiple memory segments to a selection (e.g.,Shift+Click on the timeline or the panel). Further, the user can selectmultiple memory segments via a dragged area (e.g., Shift+Click+Drag) onthe timeline, as shown in FIG. 7C. In the depicted example, the userdrags a box around a portion of the timeline to select all 8 memorysegments in the box. Further, the user interface can allow the user toselect all visible dots on the timeline. After one or more memorysegments are selected, the user can perform bulk actions on the selectedmemory segments, including favorite, export, delete, and archive, forexample, by using the user controls on control bar 852.

In some embodiments, upon selection of memory segment(s), the timelineautomatically indicate additional memory segment(s) that are related tothe selected memory segment(s), for example, by highlighting the relatedmemory segment(s) using a different color or showing the related memorysegment(s) in a different shape on the timeline (e.g., square shape inFIG. 8C). Related memory segments may have the same or similar topics,people, tags, etc., as the selected memory segment. The user interfacecan allow the user to perform bulk actions on the selected memorysegment(s) plus the related memory segments.

Collaboration Mode

As discussed above, each user in the memory retention system has his orher memory stack, which includes memory data structures such as memorychunks, memory blocks, and/or memory snippets. In some embodiments, thememory retention system allows the user to access another user's memorystack.

With reference to FIG. 7A, the user interface 700 by default providesresults based on the user's own memory stack. The user can choose to adda second memory stack by click a “+” button and specifying a seconduser, as shown by 706. In some embodiments, the user interface 700displays previously used memory stacks for easy selection. In someembodiments, the user interface 700 allows the user to reorder theselected memories stacks. In some embodiments, the user can select otherusers' memory stacks and deselect his or her own memory stack.

After the memory stacks are specified, the user's recall query isexecuted on the specified memory stacks. When displaying the results tothe recall query, the system can specify which user each resultoriginates from. In some embodiments, a result from another memory stackmay be private, and the user interface allows the user to request accessto the result. A user can choose to grant access to some or all of hisor her memory stack, which will allow anyone to use their recallinterface to add the memory stack.

In some embodiments, the recall interface provides the option to set thecollaboration mode to “parallel” or “serial.” In parallel mode, when theuser starts typing, the top recall response will be autocomplete fromthe first memory stack. The user can press <shift>, to see the firstrecall response from the second memory stack, and so on. The shiftshuffles until the max responses from each memory stack in order isshuffled through. In serial mode, when the user starts typing, the toprecall response will be autocomplete from the first memory stack. Thetop response from the second memory stack is appended to the text as afull sentence. A tab can accept one response at a time and jumps to thenext one when accepted.

FIG. 9 illustrates an exemplary recall user interface within anintegrated third-party application, in accordance with some embodiments.The memory retention system allows the user to have her thoughts, ideas,and opinions from past conversations to coalesce into suggested Twitterposts, blurbs, and team updates. Accordingly, the user can save time byreceiving suggested responses to certain tweets, and by receivingauto-complete suggestions as user composes a tweet.

In some embodiments, in the memory retention app (e.g., application654), users can choose a user account in Twitter and have the memoryretention system auto-generate responses to all new tweets from thatuser account. For each tweet, the user can view the generatedresponse(s) as shown in FIG. 9. In some embodiments, the user interfacefurther provides options to accept/shuffle/edit/cancel. The acceptoption causes the generated response to be posted in the originalthread. The shuffle option can cause a new prediction to be displayedbased on the feedback just provided. The edit option allows the user toedit the suggested post and then post. The discard option removes thepredicted post and the memory retention scores the prediction low, whichcan be used to retrain one or more of the models.

In some embodiments, the memory retention system can automaticallycreate new tweets. For example, the memory retention system can proposenew tweets based on recently generated memories in the memory stack. Thememory retention system can proactively display the new tweets and allowthe user to accept/shuffle/edit/cancel as desired.

In some embodiments, the users can have the option to connect withTwitter to fetch existing and new tweets to create memory data. Forexample, each tweet may generate one memory snippet and a twitter threadmay generate a memory block. Context data for the memory data can bederived from the metadata from the Twitter platform, such as timestampsof the tweets.

FIG. 10 illustrates an exemplary recall user interface within anintegrated third-party application, in accordance with some embodiments.The third-party application is a text editing application (e.g., GoogleDocs, Medium, Microsoft Office). While editing a document, the editorcan invoke the memory retention system, specifically, a particularuser's memory stack and transformers. In some embodiments, the memoryretention system is invoked by granting access of the document to aparticular user of the memory retention system (e.g., invitinggoogledocs@john.personal.ai to edit the Google Docs document). Inresponse to invoking the memory retention system, the system canautomatically process the existing text 1002 in the document andautomatically suggest content 1004. The content can be created byidentifying query terms (e.g., concepts, topics, entities) in thewriting 1002, retrieving memory data from the memory stack based on theidentification, and composing content based on the memory data. Thecontent can include most relevant memory data, a summary of relevantmemories, a paraphrasing of relevant memories, or a combination thereof.The composed content can conform to the writing style of the invokeduser and/or the style of the third-party application. As the editorenters additional text, the memory retention system can automaticallysuggest new content based on the additional text.

User-Specific Pods

The memory retention system ensures that user-specific memory dataremains private and secure, and that user transformers are fullycustomizable per user and available for use in real time. In someembodiments, a user-specific pod structure is allocated when a user isactively using the memory retention system (e.g., actively retaining orrecalling memory data). The user-specific pod structure is similar to aninstance of a personal computer on the cloud. Specifically, during eachuser's active session, a specific pod structure is allocated and the podstructure would be in use solely for that user—both for storage andcompute.

In some embodiments, a user pod structure is a collection ofuser-specific resources at runtime. Each pod comprises a module forproviding the recall experience and share AI experience (including therecall user interfaces), and a module for building and browsing thememory stack (including stack user interfaces and the feeds userinterfaces). In some embodiments, when the user starts an activesession, all of the modules are activated. In some embodiments, onlysome of the modules are activated, for example, because a user may beactive on feed but not active on recall. These modules can operate witha user-specific permanent storage, which stores user-specific memorydata, models, handlers, history, settings, etc. When the user ends theactive session, the memory can be deallocated and reallocated for thepod structure of another user.

In some embodiments, a user pod, similar to a computer, can be turnedoff and on or put into sleep mode. The characteristics of a user pod issimilar to a cloud resource except that the pod contains more than onecloud resource. The automation of turning on/off a pod can meetexpectations of a private and personal AI, scale to millions of users,optimize for costs, and facilitate faster development process.

In some embodiments, each compute component (e.g., transformer, search,etc.) is containerized in the pod structure; each data store component(e.g., memory stack, model) is containerized; and/or each transientstore (e.g., cache) component is containerized.

AI Intranet Protocol (“AIP”)

In some embodiments, each user in the memory retention system isassigned a unique universal address or handle. The address serves as aninterface to the user's memory data in the memory retention system. Theaddress can be in the form of “@ [unique user name] [domain name].” Forexample, John's address can be @john.hu.man.ai, and Jane's address canbe @jane.hu.man.ai. In some embodiments, the address includes additionalfields for indicating different contexts, for example, “@ [unique username] [workspace/team/company name] [domain name].” In some embodiments,the address is created when the user first signs up with the memoryretention system. During the sign-up process, the user can create aunique user name.

The user's address serves as the universal unique intranet addressendpoint accessible anywhere on the Internet to invoke variouscapabilities of the memory retention system. For example, after theaddress @john.hu.man.ai is created, https://john.hu.man.ai can beentered into a web-browser to access a website providing the memory dataof John. If the user is logged in, the website can show various userinterfaces (e.g., FIGS. 7A-7C) for recalling John's memory data. If theuser is not logged in, the website can show John's public profile andallow a user to log in, subscribe, etc.

Further, on a third-party platform, the address can be used to invokethe integrated recall functionality on the third-party platform. Forexample, on Twitter, when @john.hu.man.ai is tagged, the Twitter recallAPI can be invoked to automatically compose tweet content, as describedwith reference to FIG. 9.

In some embodiments, the memory retention system provides multipledomains for a user. For example, John can have two addressesJohn.hu.man.ai” and “john.mystack.ai.” The different domains can be usedto access different functionalities of the memory retention system. Forexample, “John.mystack.ai” can be used to browse and edit the user'smemory stack (e.g., FIGS. 8A-C) and configure user settings, while“john.hu.man.ai” can be used to access the user's public profile,perform recalls, and can be used on third-party platforms (e.g., Alexa,Chrome plug-in, etc.).

The implementation of multiple addresses allows segregation ofstorage/compute for stack needs (e.g., managing and maintaining theuser's memory stack) and recall needs. It also provides a higher levelof security, privacy, and reliability.

The operations described above are optionally implemented by componentsdepicted in FIG. 5.

FIG. 5 illustrates an example of a computing device in accordance withone embodiment. Device 500 can be a host computer connected to anetwork. Device 500 can be a client computer or a server. As shown inFIG. 5, device 500 can be any suitable type of microprocessor-baseddevice, such as a personal computer, workstation, server or handheldcomputing device (portable electronic device) such as a phone or tablet.The device can include, for example, one or more of processor 510, inputdevice 520, output device 530, storage 540, and communication device560. Input device 520 and output device 530 can generally correspond tothose described above, and can either be connectable or integrated withthe computer.

Input device 520 can be any suitable device that provides input, such asa touch screen, keyboard or keypad, mouse, or voice-recognition device.Output device 530 can be any suitable device that provides output, suchas a touch screen, haptics device, or speaker.

Storage 540 can be any suitable device that provides storage, such as anelectrical, magnetic or optical memory including a RAM, cache, harddrive, or removable storage disk. Communication device 560 can includeany suitable device capable of transmitting and receiving signals over anetwork, such as a network interface chip or device. The components ofthe computer can be connected in any suitable manner, such as via aphysical bus or wirelessly.

Software 550, which can be stored in storage 540 and executed byprocessor 510, can include, for example, the programming that embodiesthe functionality of the present disclosure (e.g., as embodied in thedevices as described above).

Software 550 can also be stored and/or transported within anynon-transitory computer-readable storage medium for use by or inconnection with an instruction execution system, apparatus, or device,such as those described above, that can fetch instructions associatedwith the software from the instruction execution system, apparatus, ordevice and execute the instructions. In the context of this disclosure,a computer-readable storage medium can be any medium, such as storage540, that can contain or store programming for use by or in connectionwith an instruction execution system, apparatus, or device.

Software 550 can also be propagated within any transport medium for useby or in connection with an instruction execution system, apparatus, ordevice, such as those described above, that can fetch instructionsassociated with the software from the instruction execution system,apparatus, or device and execute the instructions. In the context ofthis disclosure, a transport medium can be any medium that cancommunicate, propagate or transport programming for use by or inconnection with an instruction execution system, apparatus, or device.The transport readable medium can include, but is not limited to, anelectronic, magnetic, optical, electromagnetic or infrared wired orwireless propagation medium.

Device 500 may be connected to a network, which can be any suitable typeof interconnected communication system. The network can implement anysuitable communications protocol and can be secured by any suitablesecurity protocol. The network can comprise network links of anysuitable arrangement that can implement the transmission and receptionof network signals, such as wireless network connections, T1 or T3lines, cable networks, DSL, or telephone lines.

Device 500 can implement any operating system suitable for operating onthe network. Software 550 can be written in any suitable programminglanguage, such as C, C++, Java or Python. In various embodiments,application software embodying the functionality of the presentdisclosure can be deployed in different configurations, such as in aclient/server arrangement or through a Web browser as a Web-basedapplication or Web service, for example.

Although the disclosure and examples have been fully described withreference to the accompanying figures, it is to be noted that variouschanges and modifications will become apparent to those skilled in theart. Such changes and modifications are to be understood as beingincluded within the scope of the disclosure and examples as defined bythe claims.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the techniques and their practical applications. Othersskilled in the art are thereby enabled to best utilize the techniquesand various embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A computer-implemented method, comprising:obtaining a speech input of a user; obtaining a text input of the user;constructing a first instance of a memory data structure based on thespeech input, wherein the first instance comprises a transcript of thespeech input and is associated with a first set of context parameters,and wherein data associated with the first instance of the memory datastructure is obtained at least partially using a neural network;constructing a second instance of the memory data structure based on thetext input, wherein the second instance comprises at least a portion ofthe text input and is associated with a second set of contextparameters; adding the first instance and the second instance of thememory data structure to a memory stack of the user; obtaining a messagefrom a sender to the user; retrieving a particular instance of thememory data structure from the memory stack based on the message;generating a response to the message based on the retrieved particularinstance of the memory data structure; and causing display of theresponse to the message in a messaging user interface.
 2. The method ofclaim 1, wherein the messaging user interface is part of a social mediaapplication.
 3. The method of claim 1, further comprising: upongenerating the response, causing automatic transmission of the responseto the sender of the message.
 4. The method of claim 1, furthercomprising: generating a plurality of responses including the response;causing display of the plurality of responses in the messaging userinterface; receiving, via the messaging user interface, a user inputselecting the response from the plurality of responses; and causingtransmission of the response to the sender of the message.
 5. The methodof claim 1, further comprising: receiving, via the messaging userinterface, one or more edits to the response; and transmitting theedited response to the sender.
 6. The method of claim 1, furthercomprising: receiving, via the messaging user interface, a user input;responsive to the user input, generating a second response differentfrom the response; and causing display of the second response via themessaging user interface.
 7. The method of claim 1, wherein the responseis generated based on a trained machine-learning model.
 8. The method ofclaim 7, further comprising: receiving, via the messaging userinterface, a feedback of the response; and re-training themachine-learning model based on the feedback.
 9. The method of claim 1,further comprising displaying, via the messaging user interface, adescription associated with the response.
 10. The method of claim 1,wherein receiving the message from the sender comprises receiving anindication that the user is tagged in a post by the sender.
 11. Themethod of claim 1, further comprising: obtaining the first set ofcontext parameters associated with the speech input.
 12. The method ofclaim 11, wherein the first set of context parameters comprises one ormore entities extracted from the speech input.
 13. The method of claim11, wherein the first set of context parameters comprises one or moreemotions extracted from the speech input.
 14. The method of claim 11,wherein the first set of context parameters comprises a summary of thespeech input.
 15. The method of claim 11, wherein the first set ofcontext parameters comprises a concept extracted from the speech input.16. The method of claim 1, wherein the memory stack is stored in ablockchain-based database.
 17. The method of claim 1, furthercomprising: receiving, via the electronic device, an image; constructinga third instance of the memory data structure based on the image; andadding the third instance of the memory data structure to the memorystack.
 18. The method of claim 1, further comprising: receiving datafrom one or more sensors of the electronic device; constructing a fourthinstance of the memory data structure based on the received data; andadding the fourth instance of the memory data structure to the memorystack.
 19. A system, comprising: one or more processors; a memory; andone or more programs, wherein the one or more programs are stored in thememory and configured to be executed by the one or more processors, theone or more programs including instructions for: obtaining a speechinput of a user; obtaining a text input of the user; constructing afirst instance of a memory data structure based on the speech input,wherein the first instance comprises a transcript of the speech inputand is associated with a first set of context parameters, and whereindata associated with the first instance of the memory data structure isobtained at least partially using a neural network; constructing asecond instance of the memory data structure based on the text input,wherein the second instance comprises at least a portion of the textinput and is associated with a second set of context parameters; addingthe first instance and the second instance of the memory data structureto a memory stack of the user; obtaining a message from a sender to theuser; retrieving a particular instance of the memory data structure fromthe memory stack based on the message; generating a response to themessage based on the retrieved particular instance of the memory datastructure; and causing display of the response to the message in amessaging user interface.
 20. A non-transitory computer-readable storagemedium storing one or more programs, the one or more programs comprisinginstructions, which when executed by one or more processors of one ormore electronic devices, cause the electronic devices to: obtain aspeech input of a user; obtain a text input of the user; construct afirst instance of a memory data structure based on the speech input,wherein the first instance comprises a transcript of the speech inputand is associated with a first set of context parameters, and whereindata associated with the first instance of the memory data structure isobtained at least partially using a neural network; construct a secondinstance of the memory data structure based on the text input, whereinthe second instance comprises at least a portion of the text input andis associated with a second set of context parameters; add the firstinstance and the second instance of the memory data structure to amemory stack of the user; obtain a message from a sender to the user;retrieve a particular instance of the memory data structure from thememory stack based on the message; generate a response to the messagebased on the retrieved particular instance of the memory data structure;and cause display of the response to the message in a messaging userinterface.